GEDU 6170

Seminar: Focus on Research …

Becoming Informed Consumers of Education Technology Research


Instructors

Colin Lankshear (colin.lankshear@msvu.ca)
Michele Knobel (michele.knobel@msvu.ca)

Course Outline


Introduction


This course is about a particular form of critical reading and writing: namely, about how to evaluate reports of research from the standpoint of being informed research consumers.

Research reports come in different forms. Some of them are formal technical reports written at the completion of a project, such as reports made to a funding body. They can be quite long. Others are university degree theses or dissertations. Others again are articles in journals that are based on work done in a research project. Sometimes a book will report a research project or, perhaps, an edited book might bring together several chapters that each report research. In short, research reports come in different shapes and sizes. They all, however, have the same purpose – to report research for other people to consume.

Of course, just because something is published as a report of research does not make it infallible, or even good. Some research is worth taking seriously. Much of it is not. Some research is conducted well, while other research is not. The trick is to know how to tell the difference and to make savvy judgments. The aim of this course is to explain as much as possible of what the difference is, and to get us engaging in evaluation of research reports presented in journal articles.

This will involve developing a sense of the kind of activity that research is; the kinds of procedures, principles, concepts, standards and criteria that guide research as a social practice. In the time available we won’t be able get to the bottom of this, but by working in the way that we will – as teams of research consumers – we will be able to get a fair distance.

While there is not just one universally agreed on way to evaluate research there is nonetheless considerable agreement. Pretty much anyone concerned with evaluating research will, for example, agree that an acceptable report of research will contain a clear statement of research purposes and, perhaps, clearly stated research questions. They will typically agree that there needs to be a clear statement of research methodology, and that this should be appropriate for the kinds of purposes the research is addressing. There should be evidence provided that the kind of data collected for the study was of a relevant kind, and that there are good reasons to believe that the data was of good quality, and so on. These general sorts of principles and criteria and standards are widely agreed upon. Different people might disagree in particular cases about whether the data collected was, indeed, of good quality; but they will typically not disagree that quality is an important consideration.

Consequently, what we will be doing in this course is getting to grips with some of the widely agreed criteria and standards that people appeal to in evaluating research. And we will be applying these to particular pieces of research as reported in educational research literature.

We will be doing this in a way that combines individual and teamwork. More than this, when we work together in our three face-to-face seminar sessions we will be working as closely as possible to the ways that teams of researchers work when they are producing research. This is because when researchers are engaging in doing research – in producing research – they are also inevitably to some extent consuming research. They have to respond to the research that has already been done in the area they are working in, and to address aspects of that research they think are incomplete. So part of what they are doing when they develop their own research is, precisely, responding to, and bouncing off, and critiquing the research of other people. In doing this they interact with one another face to face, as far as possible or appropriate. They will bounce ideas off each other, allocate tasks, then go off and do them and come back together to see where they have all got to. They will assemble stuff, look at it, question it, decide what is to be done next, and then go off and do it, come back together again and so on. This is what we will be doing in the course. On the weekends we come together, work together, put bits and pieces together, and in the spaces in between we work individually, although we will be making use of channels of communication – like email, or internet telephony (e.g., Skype) – to keep things coordinated ahead of the next meeting. We’ll be encouraging one another, keeping in touch, and making sure that all the various “bits” that are needed for the overall team products are produced and are ready on time.

How the Course is Organized


At one level this course is organized around the three seminar weekends. Each seminar will have a specific set of purposes, described below. To fulfill these purposes it will be essential to work on certain things before and after each one – doing the relevant reading, doing some writing, staying in touch with team members, and so on.

At another level the course is organized around teams. Teams will be identified during the first weekend session, and will comprise no more than 6 members (and at least 4 members). Participants will be working partly as individuals with assigned roles and responsibilities and as members of a team.

At a third level the course is organized around the goal that in the final Saturday session, each team will present in a conference format their evaluation of a research article. They will be presenting exactly as if they were at a conference. They will have a “paper” that they present. This paper will have been assembled out of the individual contributions of the team members. They will have a digital slide slow presentation of some kind to accompany and help “carry” their presentation. They will field some questions from the floor. We will be acting and operating as much as possible like “professional researchers” who have critiqued someone else’s research and are presenting this critique as a formal conference paper.

To see how this will be done it is necessary to outline the “shape” and “purpose” of each of the three seminars.

Seminar 1 (April 18/19, 2009)

We will work all day as a whole group and as small groups discussing and getting to grips with the nature and qualities of good research as captured in our reading and, particularly in a template. During this day the teams for the duration of the course will be formed. By the end of the day the key concepts for evaluating research will have been introduced and participants will leave knowing how to use the course text to bolster their understanding of these concepts.

A copy of today's PowerPoint presentation can be found here: http://www.slideshare.com/netgrrrl

Seminar 2 (May 30/31, 2009)

Between the first and second seminar, besides doing set reading, all teams will communicate with one another to decide upon a literacy-oriented research-based article published in an academic journal within the past 5 years that will be the object of their evaluation. They will come to the second seminar with copies of that article, ready to start working on it. During the day they will work their way through elements of the template in relation to the article, considering what these elements will involve, and assigning particular “bits” to different team members. By the end of this second seminar day, every member of every team will go away with an agreed set of tasks to have completed by the time they next convene, in the third seminar.

Seminar 3 (June 27/28, 2009)

This day will be devoted to the formal presentations. Obviously, how well things go in the last part will depend heavily on the work people have done between Seminars 2 and 3, and the quality of co-ordination and communication that has occurred among team members in the interim. This may involve things like getting all members of teams on email, or on Skype, or maybe working using a collaborative writing resource like “Google Docs”, or using Google Groups for chat sessions, or whatever. But some kind of effective collaboration will be essential, and this will be one of the important “mastery outcomes” of the course.


What You Will Be Doing


Reading

The course is built around reading of three main kinds.

First, there is the short document we call the “Template” (see Appendix 1) This is the “glue” that holds the course together. It is, precisely, a framework for evaluating research.

Second, there are relevant chapters in the course textbook, A Handbook for Teacher Research. These chapters will help elucidate components of the template. That is, the various components of the template will be better grasped once they have been fleshed out by the text. This means that team members who have certain roles to perform will be “specializing” to some extent in certain chapters of the Handbook, and others will be specializing more in other chapters. But there are some chapters that everyone will need to have a good handle on.

Third, there are reports of research. We include one short report in the readings, and will use one or both of these in the first seminar – as fodder for getting to grips with some of the key concepts in the template. In addition, the various teams will locate a research-based article that they will evaluate. This will involve searching out a suitable article to evaluate (using academic search engines like http://scholar.google.com, for example), and may well involve using your MSVU online library access facility to find something suitable on a journal database.

Any participants who have wireless equipped laptops are strongly encouraged to bring them to the sessions in case we are able to establish a wireless network on the site.

Writing

You will be writing in very focused ways throughout the course. This will not be a large amount in quantitative terms, but it will be disciplined, precise, “high octane” writing of a particular kind. The emphasis will be on quality rather than quantity. The writing tasks will be very carefully structured so that you know how to do what you have to do to perform your particular role within the team process. That does not mean that it will be “easy” to do the writing. Rather, it means that it will be very clear what you have to do, but doing that clear thing will nonetheless require everyone to flex some academic muscles.

At the end of the first seminar there will be some practice writing to do based on the artifacts that have been discussed during the day. These will be specified at the time of the seminar, but they will include tasks like: “Identify the research purposes of the study in question, and discuss the extent to which you think they provide clear directions for the study”. Or, “Do the authors provide a satisfactory account of the data collection procedures and results. Do they provide grounds for confidence that the data collected was good quality data so far as the purposes of the study are concerned?”

By experimenting with different kinds of questions/tasks, participants will get a feel for what will need to be done in producing the team paper and presentation.

Communicating/collaborating

This is a crucial facet of the course. It will be absolutely essential keep in touch during the spaces between seminars, to get things coordinated, to help one another along, to get the job done. It is less important how this is done than that it is done. If people live nearby to one another they might meet face to face. Collaboration might be facilitated by phone, IM, internet chat, internet telephony, etc. A good outcome from the course would be people experimenting with unfamiliar forms of collaborating at distance with a view to finding what works best (or, alternatively, what does not work) for them.

We strongly recommend using Google Docs (http://docs.google.com) as your collaborative writing space—add us both as collaborators, and we can add in written feedback as your text progresses (our emails: michele@coatepec.net, colin@coatepec.net).

Schedule of activities

Because of the team-based nature of the course it is not possible to specify tightly in advance what individuals will be doing. This will mainly be determined during the seminars. Such detail as can be provided in advance is presented in the

Evaluation/Grading

You will need to keep up with the schedule and make sure you are prepared for the seminars. You are expected to:

• attend each of the three seminars for the entire period
• complete the appropriate readings and activities in advance of the seminars
• actively participate in the seminars
• complete the required written tasks as a team member

NOTE: Mount Saint Vincent University regulations require that all course outlines specify the following:

Statement on Plagiarism and Cheating: "University regulations on plagiarism and cheating and other academic offenses will be strictly enforced. These regulations including applicable procedures and penalties are detailed in the University Calendar and are posted on Department notice boards and on the website at http://www.msvu.ca on the Current Student's page under Academic Offenses."

Statement on correct use of language: “Correct use of language is one of the criteria included in the evaluation of all assignments.”

Penalties: “Assignments are to meet the stated deadlines. Clear evidence of full participation in each face-to-face session, and in the final written assignment needs to be demonstrated otherwise grade penalties will ensure.

Research: "Students who conduct research involving human participants must have their research reviewed in accordance with the MSVU Policies and Procedures for Ethics Review of Research Involving Humans before starting the research. Check with your course professor or Director of Graduate Education about proper procedure."

Email: Makes sure you have activated your MSVU email account, and that you check it reasonably regularly.


SEMINAR 1

Evaluating Reported Research: Introduction to Key Concepts

April 18/19, 2009

Prior reading

Prior activities

Identify some points of connection between key concepts/tasks in the template and passages in the Grinter and Eldridge paper (e.g., where they state their research questions; where they describe data collection, etc.)

Workshop activities

The day will be spent focussing on two tasks:
  • explicating the template in relation to the Grinter and Eldridge paper
  • organizing groups and identifying a focus research article to critique. In this case, each group will locate a research article to evaluate that reports a study of some aspect of education technology. Bringing a wireless laptop will greatly facilitate this search. Be sure to bring your MSVU ID card, because you need your 14-digit barcode number and the last four digits of your telephone number to access MSVU’s online library holdings.

SEMINAR 2

Working as Teams to Critique Reported Research

May 30/31, 2009

Prior reading

Everyone: A Handbook for Teacher Research, Chapters 3, 4, 5 & 6

Everyone: Read and re-read and re-read your group’s focus research article. Use the template in Appendix 1 to guide your re-readings.

Groups: Allocate additional readings in the Handbook according to roles (e.g., if you are analyzing a study that includes quantitative research, designate one group member to read Chapter 8 of the Handbook and use that to help them evaluate that dimension of the study as their contribution to the overall final paper and presentation).

Begin reading additional related texts (e.g., if your study includes grounded theory, you’ll want to read up on this approach in more detail than you’ll find in the Handbook). See the bibliography and recommended resource suggestions at the end of this study guide to help locate additional readings.

Prior activities

Email a copy of your focus research article to both Colin and Michele (Colin.Lankshear@msvu.ca, Michele.Knobel@msvu.ca). Do this as soon as you can so that we can confirm that you have indeed chosen a research article to critique.

Allocate roles and responsibilities within the group in terms of who will be contribute what to the final paper.

Begin applying the template from Seminar 1 to analyzing and critiquing your selected research study.

Begin writing up sections that will appear in your final paper.

Workshop activities

Continue analyzing, critiquing and discussing the quality of the study you are scrutinizing.

Continue writing sections that will appear in your group’s final paper for this course.

This seminar will be closer in type to a hands-on workshop than anything else, where collaborative and immediate feedback on analyses and critiques will be the order of the day. Bring laptops to help ease the composing process. Bring any additional reference texts you find as well.


SEMINAR 3

Assembling and Presenting

June 27/28, 2009

Prior reading

Relevant chapters of A Handbook for Teacher Research pertaining to your designated task within the team (e.g., if you are charged with scrutinizing data collection you would read relevant material in Chapters 9 – 12; if your task is to scrutinize data analysis you would read relevant sections in Chapters 13 – 15; if ethics of research is your focus, you might supplement reading Chapter 6 with an online paper on ethics of research; if you are evaluating the literature you would augment Chapter 5 with Chapter 7, etc.).

The research article on which your team is working (read many times until it is perfectly familiar).

Prior activities

Communication and collaboration with other members of team. “Choreograph” the presentation – who will do what and in what sequence?

Mobilize necessary resources for the presentation (e.g., PowerPoint presentation). Bring a copy on a thumb drive or CD-Rom

Have written your section of the team paper and read through the entire paper, editing and smoothing the text as you go (each of you should do this)

Print out a hard copy of your final paper, including all group members names and email addresses on a cover page, and bring to this final session.

Workshop activities

Present your team’s evaluation of your focus article, and field questions from your colleagues.

Be present for all presentations on this day.

Submit your team paper and slide presentation.



Bibliography

Grinter, R. and Eldridge, M. (2003). Wan2tlk?: Everyday text messaging. CHI 2003 Conference Proceedings. April 5-10. Fort Lauderdale, FL.

Lankshear, C. and Knobel, M (2004). A Handbook for Teacher Research. Maidenhead and New York: Open University Press.


Scholarly resources

Google Scholar (http://scholar.google.com)—click on “Scholar Preferences”, then add the MSV library (don’t forget to click “Save Preferences”. Then your searches will also show which articles can be found in the MSVU library.

Possible sources of studies to critique include:

Remember that you have full access to the MSVU library’s full archive of full-text journal articles, too.

Website: http://www.msvu.ca – then click on “library”
Username: the barcode on your MSVU ID card (a 14-digit number)
Password: Last four digits of your home phone number

OR your MSVU email username and password.

Possible sources of texts relevant to data analysis include:

• The Qualitative report: http://www.nova.edu/ssss/QR/web.html
• Kerlins.net: http://kerlins.net/bobbi/research/qualresearch/bibliography/dataanalysis.html
• Qualitative pages: http://sophia.smith.edu/~jdrisko/qualres.htm
• Quantitative research resources online: http://library.uncfsu.edu/reference/quantitative_research_websites.htm

Collaborative text production resources

• [http://docs.google.com Docs.google.com] – can be use to create collaboratively-written Word documents online
• [http://www.delicious.com Delicious.com] – an easy way to keep track of online sites you find useful, and sharing these bookmarks with your group members
• [http://notebook.google.com Notebook.Google.com] – like Delicious.com, but you can add comments and notations
• [http://www.blogger.com Blogger.com] – can be used to discuss progress of critique
• [http://www.skype.com Skype.com] – can be used to hold conference phone calls with all group members